o "Fig. 23a depicts two different hockey players set up with spherical markers. In the 
depicted view, the players are not overlapping. Fig. 23b shows the resultant circular 
reflections that will be detected by appropriate frame analysis.", with 

o "Fig. 23a depicts two different hockey players set up with prior art spherical markers. 
In the depicted view, the players are not overlapping. Fig. 23b shows the resultant 
prior art circular reflections that will be detected by appropriate frame analysis." 

• On page 43, in the first paragraph on this page, please replace: 

o "Fig. 23c depicts the same two different hockey players that were shown in Fig. 23a 
except that they are now overlapping. Fig. 23d shows the resultant circular reflections 
that will be detected in this case", with 

o "Fig. 23c depicts the same two different hockey players that were shown in Fig. 23a 
except that they are now overlapping with prior art spherical markers. Fig. 23d shows 
the resultant prior art circular reflections that will be detected in this case." 

• On page 43, in the forth paragraph on this page, please replace: 

o "Fig. 26a depicts the pre-tracking. . . " with 
o "Fig. 26a depicts the prior art pre-tracking. . . " 

With respect to my Office visit on October 2 nd , and further with respect to the referenced art of 
Sengupta, the present inventor respectfully submits that the new independent system claim 125 and 
matching method claim 141, clearly state a "automatic two-stage" system that is not found in any of the 
referenced art. The first stage is made up entirely of fixed cameras with a mandatory combined view 
covering the entire tracking area. This stage is exclusively and solely responsible for tracking all objects 
within the entire area at all times, only in two-dimensions and does not rely on any information from 
the second stage. All first cameras in this first stage work together in parallel, and not in the series or 
"hand-off' approach preferred by Sengupta. 

The second stage of claims 125 and 141 is made up entirely of moving cameras that are controlled by 
the object tracking information determined from the first stage. There video information is not required 
or used to update the tracking locations of any objects within the tracking area, which is not the case 
with Sengupta. 

With respect to new independent system claim 135 and matching method claim 151, the present 
inventor understands that these claims are acceptable to the examiner given the double patenting 
restriction and the timely filing of a terminal disclaimer in compliance with 37 CFR 1.321(c) or 
1.321(d). Please know that I intend to timely file a terminal disclaimer prior to allowance of these 
claims. I do not have this prepared as of this response because I am still researching the proper wording 
to make certain I have done this correctly. I expect to resolve this in response to your next office action. 

With respect to the Office Action: 

Starting first with points 2 and 3 of the OA, please find the revised claims 125 - 156 to have replaced 
the indefinite language of "may" with the specific language of "is." Furthermore, as per telephone 
conversations between the present inventor and the examiner, the indefinite claims language of "and / 
or" has been replaced. 
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Regarding point 4 of the OA, the present inventor intends to review this point in more detail below and 
during his upcoming office visit. If necessary, this subject will be further clarified in the final 
submission letter responding to this office action (as opposed to this draft.) 

Regarding point 5 of the OA, the present inventor refers the examiner to the specification in continued 
U.S. Patent 6,567,116 Bl, Col. 12, lines 33 through 36 that states: "Around this closed path lOp a 
minimum bounding rectangle lOr is calculated in order to quickly extract the portion of the video image 
containing the tracked object." The present inventor respectfully submits that the calculation of a 
minimum rectangle containing a tracked object in an image is equivalent to a valid estimation of that 
object's size. 

Furthermore, in Col. 14, lines 55 through 57 state: "The size of these markings will be matched to the 
target resolution of tracking cameras 24 in overhead assemblies 30c." These markings are also objects 
being tracked and their size information is captured and stored during analysis. 

Referring next to Col 1 5, lines 40 through 46 state: "Following this, the present inventors will teach 
how additional players 10 with their sticks 4 may also be simultaneously tracked and how problems 
due to object overlap, object change in size, object disappearance from view, sudden changes in 
ambient lighting and otherwise reductions in visibility for example because of fog will be handled." 
And in Col. 1 7, lines 42 through 46 state: "By knowing the expected maximum size of the object 
associated with the type of indicia found [on the] object, (e.g. pad 6 or helmet 8 or player 10), this 
process can be restricted to a maximum bounding circle in order to find the edge of the object." Both of 
these references state and imply the use of object size information. 

Regarding point 6 of the OA, the present inventor refers again to continued U.S. patent 6,567,116 Bl. 
In Col. 1 5, lines 3 1 through 40 state: "Once all overhead x-y tracking camera assemblies 30c, all 
perspective z tracking camera sets 30 and all automatic filming cameras 40 have been correctly 
installed and calibrated and all objects to be tracked have been appropriately augmented with marks 
such as 3a, 4a, 7r, 71 and 9, system 100 is ready for operation. The ensuing discussion will describe how 
the movements of a single player 10, stick 4 and puck 3 will be tracked from frame to frame during the 
duration of the event across the entire predefined field of view in the X-Y and Z dimensions." 

The present inventor notes that U.S. Patent 6,567,116 Bl discusses object tracking both in terms of X- 
Y coordinates, or alternatively expressed in polar coordinates, using a radius and angle. Both methods 
are mentioned and equivalent in their ability to represent player locations. This interchangeable aspect 
is implied by language such as in Col. 18, lines 30 through 33 that state: "First, each individual surface 
and object will have its own movements catalogued from the point it enters the field of view until at 
some later point it leaves." Here, movements may either be catalogued in X, Y (rectangular) or radius, 
angle (polar) coordinates. 

And finally, at many places through Patent 6,567,116 Bl reference is made to the X, Y tracking 
cameras and in Col. 18, lines 28 through 36 state: "Storage unit 67 will take in this continual flow of 
inked surface and corresponding object information and will create several databases for later analysis. 
First, each individual surface and object will have its own movements catalogued from the point it 
enters te field of view until at some later point it leaves. . . . Hence, to system 1 00, there is no difference 
to tracking these surfaces and objects. . ." 

And furthermore, in the present application, Fig. 14a and Fig. 14b are directed toward further re- 
explanation of the X, Y object tracking concepts first taught in continued Patent 6,567,116 Bl. 

In regard to points 7 and 8 concerning double patenting with respect to claims 121 - 124, the present 
inventor notes his intention to timely file a terminal disclaimer in compliance with 37 CFR 1.321(c) or 
1.321(d) to overcome this rejection. 
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Now, in regard to point 1, as well as points 9 and 10, the arguments provided in the applicant's 
letter dated 01/05/07 were specifically intended to address new claims 97 through 124 as well as 
cancelled claims 1 through 96. The arguments are likewise considered to be applicable to the new 
claims submitted in this response. Furthermore, these arguments cover a detailed comparison 
between Sengupta and the application as well as its claims. The applicant respectfully submits 
that these arguments' are not moot and do specifically and accurately show that Sengupta's 
invention only teaches, and therefore is limited to a single set of fixed and / or movable cameras, 
that operate in series to track and follow at least one object such as a person as this person travels 
within their combined view. In fact, Sengupta's single set of cameras is most like the present 
invention's first set of cameras (i.e. and do not include the second set of all movable cameras) 
that are also specifically arranged to track and follow at least one object such as a person as this 
person travels within their combined view. 

On the surface, Sengupta's single set of fixed and / or movable cameras might appear to be a 
better solution for tracking this single object during its travels as compared to the present 
inventor's first set of fixed cameras. The main reason being that in Sengupta's arrangement, the 
same physical number of cameras could conceptually cover movements over either a larger area, 
and / or over a more circuitous route. This is only true if the set includes at least one movable 
camera that is able to dynamically adjust its field of view therefore extending its tracking range 
over an otherwise fixed camera. Sengupta's system is ideally suited (and better suited than the 
present invention,) for tracking people as they move about in the hallways, rooms and exterior 
walkways of a building. Sengupta's system is ideally suited (and better suited than the present 
inventions,) for providing useful images to a security guard as they both track the location of the 
person of interest and view that person - this is mainly so because Sengupta's cameras are mostly 
(if not exclusively) arranged from a perspective view, which is presumed more comfortable for 
the human observer. 

However, the present inventor's intended use for tracking athletes during a sporting contest is 
significantly different from tracking individuals in a building security application. These 
differences are not handled well by Sengupta. The first major problem to address in athletics is 
tracking the players and game objects with the highest levels of confirmation. The best view for 
this function is directly over the playing surface. From a perspective view, the players will too 
often obscure and occlude each other and even more so the game object. Furthermore, if the 
present inventor were to allow moving cameras in the first set of overhead tracking cameras, than, 
while their movement could cover more total area, this same movement would both immediately 
distort the advantages of the overhead perspective and would also leave potential gaps in the total 
view space with respect to the entire playing surface (which must always be kept in full-view.) 

Sengupta's teachings continue to fail the present invention's target sports application especially 
as a large number of fast moving athletes and an even faster moving game objects begin to bunch 
up in any selected space of the playing surface. Under these very typical circumstances, any one 
of Sengupta's individual moving cameras could be exclusively viewing four, five, six or more 
players and the game object at the same moment. Each of these players and game object could 
then be going in different directions with respect to the current camera's possible range of 
tracking movement. All of which could lead to players and game objects becoming temporarily 
lost to the overall view of the system because the single moving camera was positioned to cover a 
larger space than it can accommodate at any given moment. The present invention overcomes this 
unacceptable limitation by relying exclusively upon a first set of fixed overhead cameras, whose 
views combine to contiguously and continuously cover the entire desired tracking area, i.e. the 
playing surface. Conversely, if Sengupta's entire desired tracking area is assumed to be the total 
area covered by all possible views of all cameras, and at least one camera is both movable and 
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assigned an area larger than its view can accommodate without movement, then by design, his 
system cannot maintain both contiguous and continuous views of the entire area. 

Very much unlike Sengupta, the present invention teaches that a first set of two or more fixed 
overhead cameras are fully and solely responsible, working in parallel, to provide the continuous 
X, Y location of each and every object within the combined contiguous view. Using this overall 
tracking information, a second set of exclusively movable cameras, are then automatically 
directed by the total information collected in parallel, to dynamically adjust their view to follow 
any one or more objects at any given moment. The present inventor respectfully submits that the 
functionality provided by the present inventor's approach of two distinct set of cameras, cannot 
be matched by Sengupta. To best illustrate this, the present inventor provides the following 
example setting and situations. 

The example setting is a large outside square courtyard area, where each side of the courtyard is a 
wall with a single door at midpoint leading into the courtyard. In Sengupta's simplest 
configuration, it is proposed that a single camera be fixed to cover the area surrounding each 
door. Next, it is proposed that a single movable camera is positioned on the top of any wall so 
that it can be dynamically adjusted to cover any area within the courtyard. Using this simple 
arrangement of five cameras, if any individual enters the courtyard they will be immediately 
detectable using one of the four fixed cameras. Once detected, the detecting fixed camera will 
provide its current location to the tracking system that then automatically adjust the movable 
camera to place that person in its center-of-view and then to continuously adjusts its view as the 
person enters the courtyard and moves about, essentially now out of the range of the first four 
fixed cameras solely designed to continuously watch the entrance ways. And finally, when the 
person chooses a door to leave the courtyard, the job of following the individual may be handed- 
off from the movable camera to the appropriate fixed camera covering that doorway. 

In this arrangement, in can be argued that Sengupta's system is ideal since the only possible 
access ways to the courtyard are contiguously and continuously monitored for activity while the 
larger inner space of the courtyard is optimally covered by one movable camera dynamically 
adjusting its initial view based upon a handoff signal from one of the four fixed cameras and then 
subsequently based upon its own tracking calculations. (Note that in the present invention, each 
of the second set of movable cameras never adjusts its view based upon any information it 
collects - i.e. it does not by definition contribute to the object tracking but rather responds to the 
object tracking exclusively performed by the first set of overhead cameras - these cameras do not 
self-direct.) 

In contrast, the present invention would call for a grid of fixed cameras to be hung above the 
courtyard looking straight down and together forming a contiguous and continuous view. This 
solution is obviously inferior with respect to camera and structural costs, especially as the 
courtyard expands in size. At the very least, many more cameras will end up being required. 

However, the present inventor's approach begins to take on significant advantages as additional 
complexities are presented. First, once a second person enters the courtyard either from a 
different door, or traveling in different directions from the first individual, then Sengupta's 
system will fail unless a second movable camera is added - essentially one for each person. 
Following this line of reasoning, as more and more individuals enter the courtyard, the number of 
cameras in Sengupta's approach will begin match, and even exceed the number of cameras in the 
present inventor's approach. This can be best seen by considering that ultimately, the entire 
courtyard must be continuously and contiguously viewed in order to guarantee that all individuals 
are always tracked. At this point, Sengupat's cameras are best switched to fixed cameras so that 
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their movement does not create temporary holes in the total view. Now the number of cameras 
will be near identical to the present inventors and yet their perspective view will hold a significant 
further disadvantage. Specifically, if the individuals in the courtyard begin to congregate at any 
location, their bunching can more easily hide each other or any object they hold. To compensate 
for this, Sengupta might add further moving cameras, but still not overcome this limitation. 

The present invention, on the other hand, will maintain maximum tracking capability for all 
persons and objects especially when it is understood that the courtyard is a playing surface for a 
sporting activity in which the individuals are primarily understood to maintain an upright posture 
and to keep the game object mostly at their sides or between them, as opposed to directly 
underneath their bodies. In this special set of circumstances, the overhead perspective is ideal for 
maximum tracking. This fact provides the further opportunity taught by the present inventor for 
assisting the tracking and identification of the players and game object by placing marks on their 
upper surfaces. These marks, whether they be helmet stickers or shoulder markings, are ideally 
situated for viewing by the overhead system of the present invention and not by the perspective 
arrangement of Sengupta. 

And finally with respect to Sengupta, the present invention's second set of movable cameras can 
be fully understood to perform a unique and different function than imagined or provided by the 
prior reference. Specifically, the present invention's second set of movable cameras is not 
intended, and does not in any way perform, object location tracking in the sense taught by 
Sengupta. Indeed, the present invention provides X, Y object "macro" tracking without this 
second set of movable cameras. Any given object's current X, Y centroid location within the 
playing surface is always and continuously determined by the first set of fixed overhead cameras 
and is never modified, adjusted or in any way updated or influenced by the second set of movable 
cameras. Their purpose is exclusively to provide X, Y, Z "micro" tracking of features on any 
given object, as these features move with respect to each other, still centered about the average, 
already determined X, Y centroid of their parent object. Sengupta's teaching clearly states that his 
movable cameras are used to determine the current X, Y centroid of the tracked object, and this 
by definition includes their function in the first set of macro, object tracking cameras. 

Still in regard to point 10 of the OA, as discussed in detail within the present inventor's letter 
dated August 7, 2005, the teachings of Jain et. al. reflect an entirely different approach to object 
tracking than that taught and preferred in both the present application and its continued U.S. 
patent 6,567, 1 1 6 B 1 . In summary, Jain uses multiple cameras in a "fixed volume" arrangement, 
such as discussed in Fig. 13a and Fig. 13b of the present application. This "fixed volume" 
arrangement has many problems as pointed out in regards to the figures. One of the biggest 
problems is the large number of cameras that ultimately must be used to keep the entire volume 
of potential player and game object tracking in view. By switching to the arrangement of 
overhead cameras, the present inventor significantly reduced the volume to an area equivalent to 
the playing surface. 

Furthermore, the present inventor respectfully submits that the mere arrangement of Jain's 
cameras cannot be enough to imply the teachings of the present invention. Specifically, Jain uses 
his cameras in a much different way than both the present inventor, and in fact, Sengupta. Jain 
teachings the use of multiple perspective cameras surrounding multiple objects from a perspective 
views in order to obtain a database of "voxels," or volume pixels. These "voxels" where then 
intended to be used to recreate an image of the players from any desired viewing angle, including 
angles not directly representative of the data gathering cameras. 
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The first set of tracking cameras in the present invention simply seeks to find the X, Y centroid of 
each object, not even including the Z distance off the playing surface. This X, Y centroid of each 
object is significantly less complicated to represent than a voxel surface map of any given player. 
It is also not useful for recreating views of the player's from any given angle as recreated from 
the voxels. It is more useful however, at maintaining a continuous view of all players and 
potential game objects, as well as any of their identifying marks placed on their upper surfaces. 
Any approaches to perform this same or similar function from a perspective view are significantly 
disadvantaged; at least when the players begin to bunch up as often happens team sports. 

The present inventor respectfully submits that neither Sengupta nor Jain, either separately or in 
combination, teach a two stage system of a first set of X, Y centroid tracking cameras that 
contiguously and continuously view the playing surface from an overhead perspective, followed 
by a second set of movable perspective cameras for picking up player and game object feature 
details. 

I thank you for your consideration in these matters and look forward to our upcoming meeting. 


This communication was mailed Post Office To Addressee from Lansdale, Pennsylvania on 
loll(*lol using label tK /7%543V5 2. US Examiner Senfl Behrooz. 



Sincerely, 
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